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AIRCREW  SCREENING  INSTRUMENTS  REVIEW 


Background 

This  effort  was  designed  to  identify  commercially  available  assessment 
instruments  that  may  be  useful  in  selecting  USAF  aircrew  members.  The  contract 
stipulated  that  cognitive  abilities  that  can  be  measured  by  standard  intelligence  tests  were 
not  to  be  reviewed.  Additionally,  neuropsychological  tests,  instruments  designed  for 
clinical  use,  tests  of  strictly  physiological  functions  (e.g.,  EEG),  and  instruments  designed 
for  medical  recertification  of  aircrew  members  also  were  not  included.  Instruments  and 
batteries  owned  by  foreign  governments  or  militaries  typically  were  not  examined 
because  they  are  not  commercially  available.  Experimental  instruments  developed  in 
universities  or  in  government  laboratories  also  were  not  reviewed  for  the  same  reason 
unless  the  associated  literature  indicated  that  they  had  been  commercialized. 

This  effort  also  was  restricted  to  instruments  and  batteries  cited  in  the  scientific 
literature  after  1984;  many  selection  instruments  marketed  before  this  time  have  been 
discontinued.  Finally,  the  review  was  restricted  to  instruments  in  English  or  to  non-verbal 
instruments. 


Approach 


Database  Search 

The  contract  stipulated  that  the  instruments  were  to  be  located  by  searching  the 
professional  literature.  Additionally,  the  contract  included  terms  for  the  search.  Damos 
Aviation  Services,  Inc.  (DAS)  had  access  to  three  major  databases: 

1 .  The  archive  disk  of  the  Aerospace  Medical  Association  (ASMA),  which 
covers  the  journal  from  1930  to  2002.  The  data  base  covers  association 
features,  news,  and  letters  as  well  as  research  articles.  Only  citations  published 
after  1985  as  research  articles  or  as  items  in  the  -Features”  or  -Association 
News”  sections  were  considered  in  the  searches. 

2.  The  online  database  (PsychINFO)  of  the  American  Psychological  Association 
(APA),  which  includes  over  2,150  journals  and  2.3  million  records.  Peer- 
reviewed  journals  from  1985  to  2007  were  included  in  the  search. 

3.  The  online  database  of  the  Human  Factors  and  Ergonomics  Society  (HFES), 
which  covers  all  issues  of  Human  Factors,  Ergonomics  in  Design,  and  the 
proceedings  of  the  annual  meeting. 

Using  the  tenns  included  in  the  contract  and  others,  DAS  began  a  search  of  the 
three  databases.  The  number  of  references  found  using  the  search  terms  is  shown  in  the 
columns  marked  -Hits”  in  Table  1.  All  of  the  hits  were  searched  and  those  deemed 
promising  were  retrieved  and  examined  (hand  search).  The  number  of  potential  selection 
instruments  is  recorded  under  -Tests.”  The  reader  should  note  that  the  same  selection 
instrument  could  be  recorded  multiple  times  under  -Hits.” 
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Table  1.  Search  parameters  and  results  from  three  databases. 


Search  Term 

ASMA 

APA 

HFES 

Hits 

Hand 

Search 

Tests 

Hits 

Hand 

Search 

Tests 

Hits 

Hand 

Search 

Tests 

Pilot  Selection 

236 

28 

1 

47 

11 

2 

2 

0 

0 

Personality 
Stability  test 

250 

11 

2 

49 

9 

0 

0 

0 

0 

Stable 

Personality  Test 

3 

3 

1 

25 

2 

0 

0 

0 

0 

Emotional 
Stability  test 

0 

0 

0 

184 

5 

1 

0 

0 

0 

Situational 
Awareness  test 

52 

7 

0 

11 

4 

0 

0 

0 

0 

Timesharing 
Ability  Test 

19 

11 

0 

3 

0 

0 

1 

1 

0 

Task 

Prioritization 

0 

0 

0 

0 

0 

0 

Priority  Setting 

0 

0 

0 

Dynamic  spatial 
ability 

143 

15 

1 

0 

0 

0 

Stress  Resistance 
Test 

15 

3 

2 

0 

0 

0 

Stress  Coping 

Test  with 
validity 

28 

2 

2 

0 

0 

0 

Performance 
Under  Pressure 
Test 

1 

0 

0 

0 

0 

0 

The  ASMA  database  proved  to  be  particularly  difficult  to  search;  the  disk  was 
difficult  to  load  and  the  search  was  slow.  Very  few  selection  instruments  were  located 
given  the  number  of  hits.  Because  of  the  time  limitations  on  the  contract,  the  database 
was  not  searched  for  all  the  terms. 

Selection  Instrument  References 

Two  general,  scientific  references  for  selection  instruments  were  consulted.  The 
first  was  the  Thirteenth  Mental  Measurements  Yearbook  (1998).  Entries  in  the  Yearbook 
contain  cross  references  to  tests  reviewed  in  prior  editions.  The  second  reference  was 
Tests:  A  Comprehensive  Reference  for  Assessments  in  Psychology,  Education,  and 
Business  (1997).  Both  of  these  volumes  were  searched  under  -aviation”  and  -pilot”  and 
also  — pychomotor.”  Neither  volume  produced  any  references  because,  except  for  very 
general  topics  (like  personality),  the  name  of  the  instrument  needs  to  be  known  a  priori  to 
locate  information  on  the  instrument. 
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Test  Publishers/Vendors 

As  noted  above,  searches  of  the  professional  literature  and  of  the  scientific 
references  produced  few  potential  selection  instruments.  DAS  was  aware  that  many 
potentially  useful  selection  instruments  are  not  included  in  the  scientific  literature 
because  of  proprietary  issues.  To  meet  Standard  6.9  of  the  Standards  for  Educational  and 
Psychological  Testing  (American  Educational  Research  Association,  American 
Psychological  Association,  &  National  Council  on  measurement  in  Education,  1999),  a 
test  publisher  must  keep  a  list  of  -available  studies”  pertaining  to  both  specific  and 
general  uses  of  a  specific  instrument.  Thus,  reputable  U.S.  publishers  should  have  a  list  of 
studies  that  used  their  instrument.  Canadian  and  British  publishers  also  generally  adhere 
to  this  standard. 

DAS  contacted  the  following  test  publishers/vendors:  CogScreen,  Wonderlic, 
Sigma  Assessment  Systems  (which  is  based  in  Canada  but  owned  by  Americans),  the 
Ramsey  Corporation,  Psychological  Assessment  Resources,  Inc.,  Harcourt,  Hogan 
Assessment  Systems,  IP  AT,  Aero  Innovation  (Canadian),  Symbiotics  (British),  SHL 
(British),  PsychTech  (British),  and  MHS  to  determine  if  their  reference  database 
contained  citations  pertaining  to  aircrew  selection.  Each  publisher/vendor  usually  was 
asked  if  any  of  their  tests  had  been  purchased  by  air  carriers  or  flying  schools.  The 
publisher/vendor  then  was  given  the  list  of  knowledge,  skills,  abilities,  and  other 
characteristics  (KSAOs)  from  the  contract — interpersonal  skills,  personality  stability, 
situational  awareness,  timesharing  ability,  priority  setting,  dynamic  spatial  orientation, 
and  performance  under  pressure — and  asked  to  identify  any  of  its  instruments  that 
assessed  these  KSAOs.  If  the  publisher/vendor  indicated  that  one  or  more  of  its  tests 
assessed  a  KSAO  of  interest,  it  was  asked  for  an  aviation-related  reference  list. 

The  same  basic  process  was  followed  for  two  U.S.  distributors  of  foreign  tests  that 
were  of  interest.  One  of  these  was  a  distributor  for  Harrison  Assessments,  which  is  based 
in  Hong  Kong.  The  other  was  Lafayette  Instrument  Company,  which  distributes  the 
Vienna  Test  Battery.  Both  of  these  companies  were  very  responsive.  The  amount  of 
documentation  for  the  Vienna  Test  Battery  far  exceeded  that  of  any  other  test  examined. 

A  demo  disk  also  was  provided. 

This  process  was  much  more  fruitful  than  the  previous  searches.  Generally,  the 
test  publishers/vendors  were  helpful.  Many  suggested  tests  they  felt  assessed  some  of  the 
KSAOs  and  provided  detailed  information.  The  American  exception  to  this  was 
CogScreen.  Despite  several  emails,  no  one  from  the  company  contacted  DAS.  European 
companies,  with  the  exception  of  the  British,  were  also  often  unresponsive.  The  lack  of 
response  may  be  attributed  to  the  timeframe  of  the  effort;  July  and  August  are  traditional 
vacation  months  in  Europe.  In  a  few  cases,  the  contact  person  provided  some  initial 
infonnation  and  then  failed  to  provide  the  remaining  infonnation. 

Some  vendors  would  not  release  infonnation  without  a  signature  on  a  Non 
Disclosure  Agreement  (NDA).  DAS  was  unwilling  to  sign  an  NDA  as  part  of  this 
contract  because  it  could  not  share  the  infonnation  with  the  USAF.  Consequently,  some 
infonnation  was  not  obtained. 
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Spreadsheet 

The  information  gathered  during  this  effort  is  shown  in  the  Excel  spreadsheet 
(available  upon  request  from  AFPC/DSYX  Strategic  Research  and  Assessment  Branch). 
The  first  sheet  is  labeled  -Available  Instruments”  and  is  divided  into  three  sections: 
computerized  broad-spectrum  pilot  selection  batteries,  personality  instruments,  and  other 
ability  tests.  To  be  cited  on  this  sheet,  the  vast  majority  of  the  cells  for  an  instrument  must 
be  completed,  and  the  assessed  constructs  must  fall  within  the  purview  of  the  contract. 

The  second  sheet,  — Cher  Assessments,”  describes  tests  that  DAS  believes  currently 
would  not  be  useful.  These  include  tests  owned  by  foreign  governments,  tests  in  the 
public  domain,  and  tests  assessing  constructs  not  covered  by  the  current  contract.  The 
third  sheet,  — fiture  Efforts,”  includes  companies  or  instruments  that  DAS  could  not 
locate  in  the  available  time.  Further  exploration  of  these  tests  might  be  fruitful  if  the 
USAF  decides  to  devote  more  resources  to  this  effort. 

Most  of  the  columns  are  self  explanatory,  but  several  deserve  some  comment. 

Four  columns  deal  with  cost.  Vendors  with  computer-based  tests  were  asked  to  provide 
estimates  for  a  single  workstation  and  for  three  servers  with  five  workstations  per  server. 
Many  vendors  had  difficulty  estimating  costs  with  this  type  of  configuration.  Other 
vendors  give  volume  discounts  but  were  unaccustomed  to  the  magnitude  of  a  system  like 
the  USAF.  Thus,  all  of  the  costs  in  these  columns  should  be  considered  as  rough 
approximations. 

Three  other  columns  deserve  comment.  The  first  is  — mssured  constructs.”  The 
assessed  construct  was  surprisingly  difficult  to  identify  in  several  cases  because  of  the  use 
of  unique  names.  This  problem  is  most  obvious  for  instruments  assessing  personality.  A 
developer  may  use  a  unique  name  for  proprietary  reasons  or  the  name  may  reflect  poor 
science.  Because  of  resource  limitations,  DAS  could  not  always  decide  between  these 
two  alternatives  for  a  given  instrument. 

The  second  column  needing  comment  is  adverse  impact.  Developers  rarely  assess 
adverse  impact.  However,  several  test  developers  publish  norms  by  race,  age,  and  gender. 
Such  tables  warrant  close  examination;  large  differences  in  the  norms  suggest  adverse 
impact. 

The  third  column  concerns  reliability  data.  Such  data  are  often  available  in 
published  studies.  DAS  often  did  not  have  access  to  all  of  the  studies,  so  the  spreadsheet 
may  not  reflect  the  existence  of  reliability  data  for  all  of  the  tests. 

Blank  cells  usually  indicate  that  the  vendor  did  not  supply  the  infonnation.  In 
some  cases,  a  blank  cell  indicates  that  the  infonnation  was  not  readily  available  and  DAS 
did  not  pursue  the  infonnation.  The  tenn  — IA”  in  the  cells  means  — at  applicable.”  For 
computer-based  information  and  psychomotor  batteries,  the  term  -^parallel  forms”  is 
generally  not  appropriate  because  the  stimuli  may  be  generated  randomly  for  each 
applicant  or  will  vary  according  to  the  applicant's  speed  and  number  of  conect  responses. 

Many  vendors  appear  to  have  given  very  little  thought  to  the  retesting  period. 
Many  simply  do  not  want  to  recommend  a  specific  time  period.  Most  of  the  personality 
test  vendors  believe  that  their  test  could  be  retaken  in  a  very  short  interval  because 
personality  traits  are  not  subject  to  practice  effects.  Clearly,  they  have  not  confronted  the 
organized  approach  to  obtaining  -good”  scores  that  is  common  for  aircrew  selection. 
Unless  the  vendor  specified  a  specific  retesting  period,  the  interval  is  given  as  — Mt 
listed”  in  the  spreadsheet. 
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Summary 

The  spreadsheet  should  be  considered  as  a  preliminary  assessment  of  the  available 
tools.  Several  vendors  indicated  that  they  will  have  new  selection  instruments  available 
within  a  few  months.  These  are  not  listed.  As  noted  earlier,  not  all  of  the  available 
databases  were  searched.  Indeed,  DAS’s  private  database  was  not  searched  systematically 
and  the  search  of  the  ASMA  database  should  be  completed.  Non  refereed  sources,  such 
as  proceedings,  also  should  be  searched. 

Over  the  past  approximately  10  years,  DAS  has  received  many  solicitations  for 
pilot  selection  instruments  from  Europe.  No  records  or  contact  infonnation  was  kept  of 
these  solicitations.  DAS  is  certain  that  not  all  of  these  vendors  were  contacted  for  the 
current  effort.  Additionally,  as  discussed  earlier,  some  of  the  vendors  did  not  respond  to 
multiple  emails.  Further  efforts  should  be  made  to  contact  these  companies. 
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